Batch normalization layers

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using a neural network system that includes a batch normalization layer. One of the methods includes receiving a respective first layer output for each training example in the batch; computing a plurality of normalization statistics for the batch from the first layer outputs; normalizing each component of each first layer output using the normalization statistics to generate a respective normalized layer output for each training example in the batch; generating a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and providing the batch normalization layer output as an input to the second neural network layer.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No.62/108,984, filed on Jan. 28, 2015. The disclosure of the priorapplication is considered part of and is incorporated by reference inthe disclosure of this application.

BACKGROUND

This specification relates to processing inputs through the layers ofneural networks to generate outputs.

Neural networks are machine learning models that employ one or morelayers of nonlinear units to predict an output for a received input.Some neural networks include one or more hidden layers in addition to anoutput layer. The output of each hidden layer is used as input to thenext layer in the network, i.e., the next hidden layer or the outputlayer. Each layer of the network generates an output from a receivedinput in accordance with current values of a respective set ofparameters.

SUMMARY

In general, one innovative aspect of the subject matter described inthis specification can be embodied in a neural network systemimplemented by one or more computers that includes a batch normalizationlayer between a first neural network layer and a second neural networklayer, wherein the first neural network layer generates first layeroutputs having a plurality of components, where the batch normalizationlayer is configured to, during training of the neural network system ona batch of training examples: receive a respective first layer outputfor each training example in the batch; compute a plurality ofnormalization statistics for the batch from the first layer outputs;normalize each component of each first layer output using thenormalization statistics to generate a respective normalized layeroutput for each training example in the batch; generate a respectivebatch normalization layer output for each of the training examples fromthe normalized layer outputs; and provide the batch normalization layeroutput as an input to the second neural network layer.

For a system of one or more computers to be configured to performparticular operations or actions means that the system has installed onit software, firmware, hardware, or a combination of them that inoperation cause the system to perform the operations or actions. For oneor more computer programs to be configured to perform particularoperations or actions means that the one or more programs includeinstructions that, when executed by data processing apparatus, cause theapparatus to perform the operations or actions.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. A neural network system that includes one or morebatch normalization layers can be trained more quickly than an otherwiseidentical neural network that does not include any batch normalizationlayers. For example, by including one or more batch normalization layersin the neural network system, problems caused by the distribution of agiven layer's inputs changing during training can be mitigated. This mayallow higher learning rates to be effectively used during training andmay reduce the impact of how parameters are initialized on the trainingprocess. Additionally, during training, the batch normalization layerscan act as a regularizer and may reduce the need for otherregularization techniques, e.g., dropout, to be employed duringtraining. Once trained, the neural network system that includes onenormalization layers can generate neural network outputs that are asaccurate, if not more accurate, than the neural network outputsgenerated by the otherwise identical neural network system.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example neural network system.

FIG. 2 is a flow diagram of an example process for processing an inputusing a batch normalization layer during training of the neural networksystem.

FIG. 3 is a flow diagram of an example process for processing an inputusing a batch normalization after the neural network system has beentrained.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

This specification describes a neural network system implemented ascomputer programs on one or more computers in one or more locations thatincludes a batch normalization layer.

FIG. 1 shows an example neural network system 100. The neural networksystem 100 is an example of a system implemented as computer programs onone or more computers in one or more locations, in which the systems,components, and techniques described below can be implemented.

The neural network system 100 includes multiple neural network layersthat are arranged in a sequence from a lowest layer in the sequence to ahighest layer in the sequence. The neural network system generatesneural network outputs from neural network inputs by processing theneural network inputs through each of the layers in the sequence.

The neural network system 100 can be configured to receive any kind ofdigital data input and to generate any kind of score or classificationoutput based on the input.

For example, if the inputs to the neural network system 100 are imagesor features that have been extracted from images, the output generatedby the neural network system 100 for a given image may be scores foreach of a set of object categories, with each score representing anestimated likelihood that the image contains an image of an objectbelonging to the category.

As another example, if the inputs to the neural network system 100 areInternet resources (e.g., web pages), documents, or portions ofdocuments or features extracted from Internet resources, documents, orportions of documents, the output generated by the neural network system100 for a given Internet resource, document, or portion of a documentmay be a score for each of a set of topics, with each score representingan estimated likelihood that the Internet resource, document, ordocument portion is about the topic.

As another example, if the inputs to the neural network system 100 arefeatures of an impression context for a particular advertisement, theoutput generated by the neural network system 100 may be a score thatrepresents an estimated likelihood that the particular advertisementwill be clicked on.

As another example, if the inputs to the neural network system 100 arefeatures of a personalized recommendation for a user, e.g., featurescharacterizing the context for the recommendation, e.g., featurescharacterizing previous actions taken by the user, the output generatedby the neural network system 100 may be a score for each of a set ofcontent items, with each score representing an estimated likelihood thatthe user will respond favorably to being recommended the content item.

As another example, if the input to the neural network system 100 istext in one language, the output generated by the neural network system100 may be a score for each of a set of pieces of text in anotherlanguage, with each score representing an estimated likelihood that thepiece of text in the other language is a proper translation of the inputtext into the other language.

As another example, if the input to the neural network system 100 is aspoken utterance, a sequence of spoken utterances, or features derivedfrom one of the two, the output generated by the neural network system100 may be a score for each of a set of pieces of text, each scorerepresenting an estimated likelihood that the piece of text is thecorrect transcript for the utterance or sequence of utterances.

As another example, the neural network system 100 can be part of anautocompletion system or part of a text processing system.

As another example, the neural network system 100 can be part of areinforcement learning system and can generate outputs used forselecting actions to be performed by an agent interacting with anenvironment.

In particular, each of the layers of the neural network is configured toreceive an input and generate an output from the input and the neuralnetwork layers collectively process neural network inputs received bythe neural network system 100 to generate a respective neural networkoutput for each received neural network input. Some or all of the neuralnetwork layers in the sequence generate outputs from inputs inaccordance with current values of a set of parameters for the neuralnetwork layer. For example, some layers may multiply the received inputby a matrix of current parameter values as part of generating an outputfrom the received input.

The neural network system 100 also includes a batch normalization layer108 between a neural network layer A 104 and a neural network layer B112 in the sequence of neural network layers. The batch normalizationlayer 108 is configured to perform one set of operations on inputsreceived from the neural network layer A 104 during training of theneural network system 100 and another set of operations on inputsreceived from the neural network layer A 104 after the neural networksystem 100 has been trained.

In particular, the neural network system 100 can be trained on multiplebatches of training examples in order to determine trained values of theparameters of the neural network layers. A batch of training examples isa set of multiple training examples. For example, during training, theneural network system 100 can process a batch of training examples 102and generate a respective neural network output for each trainingexample in the batch 102. The neural network outputs can then be used toadjust the values of the parameters of the neural network layers in thesequence, e.g., through conventional gradient descent andbackpropagation neural network training techniques.

During training of the neural network system 100 on a given batch oftraining examples, the batch normalization layer 108 is configured toreceive layer A outputs 106 generated by the neural network layer A 104for the training examples in the batch, process the layer A outputs 106to generate a respective batch normalization layer output 110 for eachtraining example in the batch, and then provide the batch normalizationlayer outputs 110 as an input to the neural network layer B 112. Thelayer A outputs 106 include a respective output generated by the neuralnetwork layer A 104 for each training example in the batch. Similarly,the batch normalization layer outputs 110 include a respective outputgenerated by the batch normalization layer 108 for each training examplein the batch.

Generally, the batch normalization layer 108 computes a set ofnormalization statistics for the batch from the layer A outputs 106,normalizes the layer A outputs 106 to generate a respective normalizedoutput for each training example in the batch, and, optionally,transforms each of the normalized outputs before providing the outputsas input to the neural network layer B 112.

The normalization statistics computed by the batch normalization layer108 and the manner in which the batch normalization layer 108 normalizesthe layer A outputs 106 during training depend on the nature of theneural network layer A 104 that generates the layer A outputs 106.

In some cases, the neural network layer A 104 is a layer that generatesan output that includes multiple components indexed by dimension. Forexample, the neural network layer A 104 may be a fully-connected neuralnetwork layer. In some other cases, however, the neural network layer A104 is a convolutional layer or other kind of neural network layer thatgenerates an output that includes multiple components that are eachindexed by both a feature index and a spatial location index. Generatingthe batch normalization layer output during training of the neuralnetwork system 100 in each of these two cases is described in moredetail below with reference to FIG. 2.

Once the neural network system 100 has been trained, the neural networksystem 100 may receive a new neural network input for processing andprocess the neural network input through the neural network layers togenerate a new neural network output for the input in accordance withthe trained values of the parameters of the components of the neuralnetwork system 100. The operations performed by the batch normalizationlayer 108 during the processing of the new neural network input alsodepend on the nature of the neural network layer A 104. Processing a newneural network input after the neural network system 100 has beentrained is described in more detail below with reference to FIG. 3.

The batch normalization layer 108 may be included at various locationsin the sequence of neural network layers and, in some implementations,multiple batch normalization layers may be included in the sequence.

In the example of FIG. 1, in some implementations, the neural networklayer A 104 generates outputs by modifying inputs to the layer inaccordance with current values of a set of parameters for the firstneural network layer, e.g., by multiplying the input to the layer by amatrix of the current parameter values. In these implementations, theneural network layer B 112 may receive an output from the batchnormalization layer 108 and generate an output by applying a non-linearoperation, i.e., a non-linear activation function, to the batchnormalization layer output. Thus, in these implementations, the batchnormalization layer 108 is inserted within a conventional neural networklayer, and the operations of the conventional neural network layer aredivided between the neural network layer A 104 and the neural networklayer B 112.

In some other implementations, the neural network layer A 104 generatesthe outputs by modifying layer inputs in accordance with current valuesof a set of parameters to generate a modified first layer inputs andthen applying a non-linear operation to the modified first layer inputsbefore providing the output to the batch normalization layer 108. Thus,in these implementations, the batch normalization layer 108 is insertedafter a conventional neural network layer in the sequence.

FIG. 2 is a flow diagram of an example process 200 for generating abatch normalization layer output during training of a neural network ona batch of training examples. For convenience, the process 200 will bedescribed as being performed by a system of one or more computerslocated in one or more locations. For example, a batch normalizationlayer included in a neural network system, e.g., the batch normalizationlayer 108 included in the neural network system 100 of FIG. 1,appropriately programmed, can perform the process 200.

The batch normalization layer receives lower layer outputs for the batchof training examples (step 202). The lower layer outputs include arespective output generated for each training example in the batch bythe layer below the batch normalization layer in the sequence of neuralnetwork layers.

The batch normalization layer generates a respective normalized outputfor each training example in the batch (step 204). That is, the batchnormalization layer generates a respective normalized output from eachreceived lower layer output.

In some cases, the layer below the batch normalization layer is a layerthat generates an output that includes multiple components indexed bydimension.

In these cases, the batch normalization layer computes, for eachdimension, the mean and the standard deviation of the components of thelower layer outputs that correspond to the dimension. The batchnormalization layer then normalizes each component of each of the lowerlevel outputs using the means and standard deviations to generate arespective normalized output for each of the training examples in thebatch. In particular, for a given component of a given output, the batchnormalization layer normalizes the component using the mean and thestandard deviation computed for the dimension corresponding to thecomponent. For example, in some implementations, for a component x_(k,i)corresponding to the k-th dimension of the i-th lower layer output froma batch β, the normalized output {circumflex over (x)}_(k,i) satisfies:

${{\hat{x}}_{k,i} = \frac{x_{k,i} - \mu_{B}}{\sigma_{B}}},$

where μ_(B) is the mean of the components corresponding to the k-thdimension of the lower layer outputs in the batch β and σ_(B) is thestandard deviation of the components corresponding to the k-th dimensionof the lower layer outputs in the batch β. In some implementations, thestandard deviation is a numerically stable standard deviation that isequal to (σ_(B) ²+ε)^(1/2), where ε is a constant value and σ_(B) ² isthe variance of the components corresponding to the k-th dimension ofthe lower layer outputs in the batch β.

In some other cases, however, the neural network layer below the batchnormalization layer is a convolutional layer or other kind of neuralnetwork layer that generates an output that includes multiple componentsthat are each indexed by both a feature index and a spatial locationindex.

In some of these cases, the batch normalization layer computes, for eachpossible feature index and spatial location index combination, the meanand the variance of the components of the lower layer outputs that havethat feature index and spatial location index. The batch normalizationlayer then computes, for each feature index, the average of the meansfor the feature index and spatial location index combinations thatinclude the feature index. The batch normalization layer also computes,for each feature index, the average of the variances for the featureindex and spatial location index combinations that include the featureindex. Thus, after computing the averages, the batch normalization layerhas computed a mean statistic for each feature across all of the spatiallocations and a variance statistic for each feature across all of thespatial locations.

The batch normalization layer then normalizes each component of each ofthe lower level outputs using the average means and the averagevariances to generate a respective normalized output for each of thetraining examples in the batch. In particular, for a given component ofa given output, the batch normalization layer normalizes the componentusing the average mean and the average variance for the feature indexcorresponding to the component, e.g., in the same manner as describedabove when the layer below the batch normalization layer generatesoutputs indexed by dimension.

In others of these cases, the batch normalization layer computes, foreach feature index the mean and the variance of the components of thelower layer outputs that correspond to the feature index, i.e., thathave the feature index.

The batch normalization layer then normalizes each component of each ofthe lower level outputs using the means and the variances for thefeature indices to generate a respective normalized output for each ofthe training examples in the batch. In particular, for a given componentof a given output, the batch normalization layer normalizes thecomponent using the mean and the variance for the feature indexcorresponding to the component, e.g., in the same manner as describedabove when the layer below the batch normalization layer generatesoutputs indexed by dimension.

Optionally, the batch normalization layer transforms each component ofeach normalized output (step 206).

In cases where the layer below the batch normalization layer is a layerthat generates an output that includes multiple components indexed bydimension, the batch normalization layer transforms, for each dimension,the component of each normalized output in the dimension in accordancewith current values of a set of parameters for the dimension. That is,the batch normalization layer maintains a respective set of parametersfor each dimension and uses those parameters to apply a transformationto the components of the normalized outputs in the dimension. The valuesof the sets of parameters are adjusted as part of the training of theneural network system. For example, in some implementations, thetransformed normalized output y_(k,i) generated from the normalizedoutput {circumflex over (x)}_(k,i) satisfies:

y _(k,i)=γ_(k) +A _(k),

where γ_(k) and A_(k) are the parameters for the k-th dimension.

In cases where the layer below the batch normalization layer is aconvolutional layer, the batch normalization layer transforms, for eachcomponent of each of the normalized outputs, the component in accordancewith current values of a set of parameters for the feature indexcorresponding to the component. That is, the batch normalization layermaintains a respective set of parameters for each feature index and usesthose parameters to apply a transformation to the components of thenormalized outputs that have the feature index, e.g., in the same manneras described above when the layer below the batch normalization layergenerates outputs indexed by dimension. The values of the sets ofparameters are adjusted as part of the training of the neural networksystem.

The batch normalization layer provides the normalized outputs or thetransformed normalized outputs as input to a layer above the batchnormalization layer in the sequence (step 208).

After the neural network has generated the neural network outputs forthe training examples in the batch, the normalization statistics arebackpropagated through as part of adjusting the values of the parametersof the neural network, i.e., as part of performing the backpropagationtraining technique.

FIG. 3 is a flow diagram of an example process 300 for generating abatch normalization layer output for a new neural network input afterthe neural network has been trained. For convenience, the process 300will be described as being performed by a system of one or morecomputers located in one or more locations. For example, a batchnormalization layer included in a neural network system, e.g., the batchnormalization layer 108 included in the neural network system 100 ofFIG. 1, appropriately programmed, can perform the process 300.

The batch normalization layer receives a lower layer output for the newneural network input (step 302). The lower layer output is an outputgenerated for the new neural network input by the layer below the batchnormalization layer in the sequence of neural network layers.

The batch normalization layer generates a normalized output for the newneural network input (step 304).

If the outputs generated by the layer below the batch normalizationlayer are indexed by dimension, the batch normalization layer normalizeseach component of the lower layer output using pre-computed means andstandard deviations for each of the dimensions to generate a normalizedoutput. In some cases, the means and standard deviations for a givendimension are computed from the components in the dimension of all ofoutputs generated by the layer below the batch normalization layerduring the training of the neural network system.

In some other cases, however, the means and standard deviations for agiven dimension are computed from the components in the dimension of thelower layer outputs generated by the layer below the batch normalizationlayer after training, e.g., from lower layer outputs generated during ina most recent time window of specified duration or from a specifiednumber of lower layer outputs most recently generated by the layer belowthe batch normalization layer.

In particular, in some cases the distribution of network inputs and,accordingly, the distribution of lower layer outputs may change betweenthe training examples used during training and the new neural networkinputs used after the neural network system is trained, e.g., if the newneural network inputs are different kinds of inputs from the trainingexamples. For example, the neural network system may have been trainedon user images and may now be used to process video frames. The userimages and the video frames likely have different distributions in termsof the classes pictured, image properties, composition, and so on.Therefore, normalizing the lower layer inputs using statistics from thetraining may not accurately capture the statistics of the lower layeroutputs being generated for the new inputs. Thus, in these cases, thebatch normalization layer can use normalization statistics computed fromlower layer outputs generated by the layer below the batch normalizationlayer after training.

If the outputs generated by the layer below the batch normalizationlayer are indexed by feature index and spatial location index, the batchnormalization layer normalizes each component of the lower layer outputusing pre-computed average means and average variances for each of thefeature indices, to generate a normalized output. In some cases, asdescribed above, the average means and average variances for a givenfeature index, are computed from the outputs generated by the layerbelow the batch normalization layer for all of the training examplesused during training. In some other cases, as described above, the meansand standard deviations for a given feature index are computed from thelower layer outputs generated by the layer below the batch normalizationlayer after training.

Optionally, the batch normalization layer transforms each component ofthe normalized output (step 306).

If the outputs generated by the layer below the batch normalizationlayer are indexed by dimension, the batch normalization layertransforms, for each dimension, the component of the normalized outputin the dimension in accordance with trained values of the set ofparameters for the dimension. If the outputs generated by the layerbelow the batch normalization layer are indexed by feature index andspatial location index, the batch normalization layer transforms eachcomponent of the normalized output in accordance with trained values ofthe set of parameters for the feature index corresponding to thecomponent.

The batch normalization layer provides the normalized output or thetransformed normalized output as input to the layer above the batchnormalization layer in the sequence (step 308).

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them.

A computer program (which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code) can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Computers suitable for the execution of a computer program include, byway of example, can be based on general or special purposemicroprocessors or both, or any other kind of central processing unit.Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essentialelements of a computer are a central processing unit for performing orexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few. Computer readablemedia suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto optical disks; and CD ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A neural network system implemented by one ormore computers, the neural network system comprising: a batchnormalization layer between a first neural network layer and a secondneural network layer, wherein the first neural network layer generatesfirst layer outputs having a plurality of components, and wherein thebatch normalization layer is configured to, during training of theneural network system on a batch of training examples: receive arespective first layer output for each training example in the batch;compute a plurality of normalization statistics for the batch from thefirst layer outputs; normalize each component of each first layer outputusing the normalization statistics to generate a respective normalizedlayer output for each training example in the batch; generate arespective batch normalization layer output for each of the trainingexamples from the normalized layer outputs; and provide the batchnormalization layer output as an input to the second neural networklayer.
 2. The neural network system of claim 1, wherein the plurality ofcomponents of the first layer output are indexed by dimension, andwherein computing a plurality of normalization statistics for the firstlayer outputs comprises: computing, for each of the dimensions, a meanof the components of the first layer outputs in the dimension; andcomputing, for each of the dimensions, a standard deviation of thecomponents of the first layer outputs in the dimension.
 3. The neuralnetwork system of claim 2, wherein normalizing each component of eachlayer output comprises: normalizing the component using the computedmean and computed standard deviation for the dimension corresponding tothe component.
 4. The neural network system of claim 2, whereingenerating the respective batch normalization layer output for each ofthe training examples from the normalized layer outputs comprises:transforming, for each dimension, the component of the normalized layeroutput for the training example in the dimension in accordance withcurrent values of a set of parameters for the dimension.
 5. The neuralnetwork system of claim 4, wherein the batch normalization layer isconfigured to, after the neural network system has been trained todetermine trained values of the parameters for each of the dimensions:receive a new first layer output generated by the first neural networklayer for a new neural network input; normalize each component of thenew first layer output using pre-computed mean and standard deviationstatistics for the dimensions to generate a new normalized layer output;generate a new batch normalization layer output by transforming, foreach dimension, the component of the new normalized layer output for thetraining example in the dimension in accordance with trained values ofthe set of parameters for the dimension; and provide the batchnormalization layer output as a new layer input to the second neuralnetwork layer.
 6. The neural network system of claim 5, wherein thepre-computed mean and standard deviation statistics for the dimensionsare computed from first layer outputs generated by the first neuralnetwork layer during training of the neural network system.
 7. Theneural network system of claim 5, wherein the pre-computed mean andstandard deviation statistics for the dimensions are computed from newfirst layer outputs generated by the first neural network layer afterthe neural network has been trained.
 8. The neural network system ofclaim 7, wherein new neural network inputs processed by the neuralnetwork system after the neural network system has been trained are adifferent type of input than the training examples used to train theneural network system.
 9. The neural network system of claim 1, whereinthe first neural network layer is a convolutional layer, wherein theplurality of components of the first layer output are indexed by featureindex and spatial location index, and wherein computing a plurality ofnormalization statistics for the first layer outputs comprises:computing, for each combination of feature index and spatial locationindex, a mean of the components of the first layer outputs having thefeature index and spatial location index; computing, for each featureindex, an average of the means for combinations that include the featureindex; computing, for each combination of feature index and spatiallocation index, a variance of the components of the first layer outputshaving the feature index and spatial location index; and computing, foreach feature index, an average of the variances for combinations thatinclude the feature index.
 10. The neural network system of claim 9,wherein normalizing each component of each layer output comprises:normalizing the component using the average of the means and the averageof the variances for the feature index corresponding to the component.11. The neural network system of claim 9, wherein generating therespective batch normalization layer output for each of the trainingexamples from the normalized layer outputs comprises: transforming eachcomponent of the normalized layer output in accordance with currentvalues of a set of parameters for the feature index corresponding to thecomponent.
 12. The neural network system of claim 11, wherein the batchnormalization layer is configured to, after the neural network has beentrained to determine trained values of the parameters for each of thedimensions: receive a new first layer input generated from a new neuralnetwork input; normalize each component of the new first layer outputusing pre-computed mean and standard deviation statistics for thefeature indices to generate a new normalized layer output; generate anew batch normalization layer output by transforming each component ofthe normalized layer output in accordance with trained values of the setof parameters for the feature index corresponding to the component; andprovide the new batch normalization layer output as a new layer input tothe second neural network layer.
 13. The neural network system of claim1, wherein the first neural network layer is a convolutional layer,wherein the plurality of components of the first layer output areindexed by feature index and spatial location index, and whereincomputing a plurality of normalization statistics for the first layeroutputs comprises, for each of the feature indices: computing a mean ofthe components of the first layer outputs that correspond to the featureindex; and computing a variance of the components of the first layeroutputs that correspond to the feature index.
 14. The neural networksystem of claim 13, wherein normalizing each component of each layeroutput comprises: normalizing the component using the mean and thevariance for the feature index corresponding to the component.
 15. Theneural network system of claim 13, wherein generating the respectivebatch normalization layer output for each of the training examples fromthe normalized layer outputs comprises: transforming each component ofthe normalized layer output in accordance with current values of a setof parameters for the feature index corresponding to the component. 16.The neural network system of claim 15, wherein the batch normalizationlayer is configured to, after the neural network has been trained todetermine trained values of the parameters for each of the dimensions:receive a new first layer input generated from a new neural networkinput; normalize each component of the new first layer output usingpre-computed mean and standard deviation statistics for the featureindices to generate a new normalized layer output; generate a new batchnormalization layer output by transforming each component of thenormalized layer output in accordance with trained values of the set ofparameters for the feature index corresponding to the component; andprovide the new batch normalization layer output as a new layer input tothe second neural network layer.
 17. The neural network system of claim1, wherein the first neural network layer generates the first layeroutputs by modifying first layer inputs in accordance with currentvalues of a set of parameters for the first neural network layer. 18.The neural network system of claim 17, wherein the second neural networklayer generates second layer outputs by applying a non-linear operationto the batch normalization layer outputs.
 19. The neural network systemof claim 1, wherein the first neural network layer generates the firstlayer outputs by modifying first layer inputs in accordance with currentvalues of a set of parameters to generate modified first layer inputsand then applying a non-linear operation to the modified first layerinputs.
 20. The neural network system of claim 1, wherein, during thetraining of the neural network, the neural network system is configuredto backpropagate the normalization statistics as part of adjustingvalues of parameters of the neural network.
 21. A method performed byone or more computers implementing a batch normalization layer that isbetween a first neural network layer and a second neural network layerin a neural network, wherein the first neural network layer generatesfirst layer outputs having a plurality of components, and wherein themethod comprises, during training of the neural network on a batch oftraining examples: receiving a respective first layer output for eachtraining example in the batch; computing a plurality of normalizationstatistics for the batch from the first layer outputs; normalizing eachcomponent of each first layer output using the normalization statisticsto generate a respective normalized layer output for each trainingexample in the batch; generating a respective batch normalization layeroutput for each of the training examples from the normalized layeroutputs; and providing the batch normalization layer output as an inputto the second neural network layer.
 22. One or more non-transitorycomputer storage media encoded with a computer program, the computerprogram comprising instructions that when executed by one or morecomputers cause the one or more computers to perform operations toimplement a neural network, the neural network comprising: a batchnormalization layer between a first neural network layer and a secondneural network layer, wherein the first neural network layer generatesfirst layer outputs having a plurality of components, and wherein thebatch normalization layer is configured to, during training of theneural network system on a batch of training examples: receive arespective first layer output for each training example in the batch;compute a plurality of normalization statistics for the batch from thefirst layer outputs; normalize each component of each first layer outputusing the normalization statistics to generate a respective normalizedlayer output for each training example in the batch; generate arespective batch normalization layer output for each of the trainingexamples from the normalized layer outputs; and provide the batchnormalization layer output as an input to the second neural networklayer.